From Appearance to Essence: Comparing Truth Discovery Methods without Using Ground Truth

Truth discovery has been widely studied in recent years as a fundamental means for resolving the con icts in multi-source data. Although many truth discovery methods have been proposed based on di erent considerations and intuitions, investigations show that no single method consistently outperforms the others. To select the right truth discovery method for a speci c application scenario, it becomes essential to evaluate and compare the performance of di erent methods. A drawback of current research e orts is that they commonly assume the availability of certain ground truth for the evaluation of methods. However, the ground truth may be very limited or even out-of-reach in practice, rendering the evaluation biased by the small ground truth or even unfeasible. In this paper, we present CompTruthHyp, a general approach for comparing the performance of truth discovery methods without using ground truth. In particular, our approach calculates the probability of observations in a dataset based on the output of di erent methods. The probability is then ranked to re ect the performance of these methods. We review and compare twelve existing truth discovery methods and consider both single-valued and multi-valued objects. Empirical studies on both real-world and synthetic datasets demonstrate the e ectiveness of our approach for comparing truth discovery methods.